Skip to content

Ship the quantized kernels as their own library - #21642

Open
shoumikhin wants to merge 24 commits into
gh/shoumikhin/92/headfrom
gh/shoumikhin/93/head
Open

Ship the quantized kernels as their own library#21642
shoumikhin wants to merge 24 commits into
gh/shoumikhin/92/headfrom
gh/shoumikhin/93/head

Conversation

@shoumikhin

@shoumikhin shoumikhin commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

The problem

The wheel now ships the runtime, the CPU kernels, the XNNPACK delegate, the thread pool and the
profiler as separate shared libraries a C++ application can link. The quantized kernels are the
one piece from that set still missing.

They exist today only inside libquantized_ops_aot_lib.so, which sits beside the Python bindings
and is built for ahead-of-time export. A C++ application that loads a quantized .pte has no
library to link, so it fails at run time:

Missing operator: [0] quantized_decomposed::dequantize_per_tensor.out

The change

Builds the quantized kernels as a shared library, ships it beside the others, and names it as a
component.

executorch/lib/libexecutorch_kernels_quantized.so
find_package(executorch REQUIRED COMPONENTS kernels_quantized)
target_link_libraries(my_app PRIVATE executorch::runtime
                                     executorch::kernels_quantized)

Named after what it provides rather than after the target that produces it, matching
libexecutorch_kernels_optimized.so next to it.

Two things this needed that were not obvious. The library is named explicitly in the wheel's
build target list, because a C++ application is its only consumer, and without naming it the
target is generated and never built. And the single-owner checks skip the ahead-of-time
libraries, which deliberately carry their own copy because export runs in a process that never
loads the runtime libraries a C++ application links. They are recognised by linking torch rather
than by name, so a rename does not silently drop the exclusion.

Before and after

BEFORE                                  AFTER

a quantized .pte from C++               a quantized .pte from C++
  Missing operator:                       runs, matching eager PyTorch
  quantized_decomposed::
  dequantize_per_tensor.out

nothing to link                         target_link_libraries(app PRIVATE
                                          executorch::kernels_quantized)

Nothing changes for a Python user. The export path keeps using the ahead-of-time library.

Test plan

  • exactly one shipped object defines the quantized operators, and it is this library
  • a C++ application linking executorch::kernels_quantized exports a quantized model, runs it,
    and matches eager PyTorch

The comparison tolerance is per model rather than one number for all: a correct int8 run differs
from eager PyTorch by about one quantization step, which for this model is 5e-3, while a float32
model matches to within rounding. Using the looser number for both would stop the float path
catching a real regression.

Ran on Linux x86_64 and aarch64 against a wheel built from source and installed into a clean
environment. The quantized model matches eager PyTorch to 4.8e-03, one quantization step.

[ghstack-poisoned]
@shoumikhin

shoumikhin commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

@pytorch-bot

pytorch-bot Bot commented Aug 7, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21642

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 7, 2026
@shoumikhin shoumikhin added ciflow/periodic ciflow/trunk ciflow/binaries ciflow/binaries/all Release PRs with this label will build wheels for all python versions ciflow/nightly ciflow/cuda labels Aug 7, 2026
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
[ghstack-poisoned]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/binaries/all Release PRs with this label will build wheels for all python versions ciflow/binaries ciflow/cuda ciflow/nightly ciflow/periodic ciflow/trunk CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant